Image display device

ABSTRACT

An object is to provide an image display device capable of correcting a position in a simple manner even when a deviation occurs in position alignment in accordance with movement of a user. The image display device  100  includes a correction processing unit  102  that corrects a deviation when the deviation occurs between image data G and a steel frame T in a field of vision of a user U that occurs in accordance with movement of the user. This correction processing unit  102  acquires a coordinates correction position S that becomes a reference position used for a correction process from the steel frame T 1  or the like that is a partial member using design data stored by a storage unit  103  and a self-position of the user U with respect to the steel frame T. The correction processing unit  102  performs correction by performing position alignment of the image data G on the basis of the one coordinates correction position S.

TECHNICAL FIELD

The present invention relates to an image display device that uses acomposite reality space (mixed reality (MR)).

BACKGROUND ART

Composing an image with an object present in a reality space using MRtechnologies is known. For example, in Patent Literature 1, overlappingrepresentation of a reality-space image and a virtual-space image isdescribed.

CITATION LIST Patent Literature

-   [Patent Literature 1] Japanese Unexamined Patent Publication No.    2008-293209

SUMMARY OF INVENTION Technical Problem

In recent years, enabling easy perception of a finished product bycomposing image data of decorative materials and the like withstructures of only frames, beams, or the like in a construction stage ofa building by using MR technologies has been conceived.

However, a case in which it is difficult to perform position alignmentof the image data in a structure frequently occurs. Generally, it isconceivable to perform position alignment of image data by attaching amarker to a structure. In that case, when a user wearing gogglesenabling viewing in an MR space such as MR goggles moves by moving awayfrom a structure to which a marker is attached, a deviation in theposition alignment between the structure and the image data occurs. Forthis reason, it may be conceivable to dispose markers corresponding topositions throughout the structure, but much labor would be required forsetting and disposing such markers corresponding to the positions.

Thus, in order to solve the problems described above, an object is toprovide an image display device capable of correcting a deviation inposition alignment, which occurs when a user moves in a composite space,in a simple manner.

Solution to Problem

According to the present invention, there is provided an image displaydevice that composes and displays a predetermined image for acomposition target object present in a reality space, the image displaydevice including: a storage unit configured to store design datarepresenting a positional relation between parts of the compositiontarget object; a display unit configured to generate and displays acomposed image acquired by composing a predetermined image using onepart of the composition target object as a reference position; and acorrection unit configured to correct a deviation between thepredetermined image and the composition target object that occurs inaccordance with movement of a user, in which the correction unitacquires a coordinates correction position that becomes a referenceposition used for a correction process in the one part of thecomposition target object using the design data and a self-position ofthe user with respect to the composition target object and corrects acomposition position of the predetermined image on the basis of thecoordinates correction position.

Advantageous Effects of Invention

According to the present invention, a deviation in position alignment ina composite reality space can be corrected in a simple manner.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a view of steel frames T that are structuresaccording to this embodiment and a view acquired by composing image dataG of columns with the steel frames T.

FIG. 2 is an explanatory diagram of coordinates correction positions.

FIG. 3 is an explanatory diagram illustrating a composition processusing the coordinates correction positions.

FIG. 4 is a block diagram illustrating a functional configuration of animage display device 100.

FIG. 5 is a flowchart illustrating operations of the image displaydevice 100.

FIG. 6 is a flowchart illustrating detailed operations of a processS101.

FIG. 7 is a flowchart illustrating detailed processes of a process ofselecting a self-position/posture estimation result and a correctionprocess performed by a correction processing unit 102.

FIG. 8 is a flowchart illustrating a process of selecting a coordinatescorrection position to be used from coordinates correction positioncandidates.

FIG. 9 is a flowchart illustrating a process of calculating a deviationof a coordinates correction position that is performed by a correctionprocessing unit 102.

FIG. 10 is a flowchart illustrating another process performed when apositional deviation between image data G and a steel frame T iscalculated.

FIG. 11 is a diagram illustrating an example of a hardware configurationof an image display device 100 according to one embodiment of thepresent disclosure.

DESCRIPTION OF EMBODIMENTS

An embodiment of the present invention will be described with referenceto the attached drawings. If possible, the same reference signs will beassigned to the same parts, and duplicate description will be omitted.

FIG. 1 illustrates a view of steel frames T that are structuresaccording to this embodiment and a view acquired by composing image dataG of decorative materials indicating columns with the steel frames T.FIG. 1 illustrates a view that can be seen by a user wearing MR gogglesthat are an image display device according to this embodiment. FIG. 1(a)is a view illustrating steel frames T that are structures present in areality space before composition of image data G of decorativematerials. In FIG. 1 , the steel frames T are composed of steel frames Tto Tn. In FIG. 1 , although steel frames up to a steel frame T9 areillustrated, it is apparent that steel frames up to a steel frame Tn(here, n is a natural number) may be present. This similarly appliesalso to image data Gn (here, n is a natural number; for example, G1 toG9).

In this embodiment, although columns are formed as steel frames, thecolumns are not limited to steel frames and may be concrete columns ormay be in a state of being covered with decorative materials. An imagethat is composed and displayed is not limited to an image of adecorative material and may be configured as an image of a pipearrangement, a wiring, furniture, or the like.

FIG. 1(b) is a view representing a state in which actual steel frames Tare covered with decorative materials used for decoration and is a viewacquired when image data G representing the pieces of decorativematerial is composed with the steel frames T in a composite realityspace. Design data indicating positions, sizes, and the like of thesteel frames T1 to Tn and image data G1 to Gn of the pieces ofdecorative material are stored in advance. The image data G is athree-dimensional image composed of the image data G1 to Gn. The imagedata G1 to Gn of a plurality of decorative materials are associated inaccordance with the positions of the steel frames T1 to Tn in advance.

A correction correspondence part corresponding to a coordinatescorrection position to be described below is set in each piece of imagedata Gn. For example, when composition is performed with a correctioncorrespondence part of the image data G1 of a decorative material beingaligned with the coordinates correction position set in the steel frameT1, image data G2 of another decorative material is also aligned inposition with a corresponding steel frame T2. Similarly, the image dataGn and the like of other pieces of decorative materials are aligned inposition with corresponding steel frames Tn. However, an error may beincluded in the alignment in position of correction correspondence partsof the image data G2 and the like with the coordinates correctionpositions set in the steel frame T2 and the like.

When the steel frames T and the image data G of the decorative materialof the steel frames T are initially aligned in position, for example, aninitial marker used for the image data G1 is attached to a predeterminedplace on the steel frame T1, whereby position alignment is performed.The image data G2 and the like configure image data G of which a mutualpositional relation is fixed, and thus the other steel frames T2 and thelike are automatically aligned in position. In addition, the image dataG is a three-dimensional image, and thus, this image data G is rotatedin accordance with a direction in which a user sees the steel frame T.In this way, by performing position alignment on the basis of theinitial marker, position alignment between all the steel frames T1 to Tnand the image data G1 to Gn can be performed.

The position alignment will be additionally described in detail. FIG. 2is an explanatory diagram of coordinates correction positions. In FIG. 1, performing only position alignment is shown. As illustrated in FIG. 2, the steel frame T1 has faces in four directions. The positionalignment of the image data G1 of the decorative material is performedusing one of the faces of the steel frame T as a reference. Thecoordinates correction position represents a reference position used forposition alignment between the position of the steel frame T and theimage data G of the decorative material.

FIG. 2(a) is a schematic view acquired when viewing the upper face ofthe steel frame T. In this embodiment, this steel frame T is a prismhaving an intersection C of base lines at the center thereof, and across-section thereof is a tetrahedron. Here, the steel frame T1 will befocused on. In faces of the steel frame T1, coordinates correctionpositions 511 to S14 are set.

When a user moves while viewing the steel frame T1, deviations inpositions between other steel frames T2 to T4 (the decorative materialsK2 to K4) and the image data G2 to G4 of the decorative materials occur.For this reason, in a case in which deviations occur, the deviationsneed to be corrected. In other words, when a user moves while seeing thesteel frame T1, the reference marker that has been initially setdisappears from the field of view, and tracking of the position/angleusing this reference marker cannot be performed, and tracking usingself-position/posture estimation using SLAM is performed. In thisself-position/posture estimation, an estimation error occurs inaccordance with movement.

In this embodiment, by identifying a steel frame T that is the nearestfrom a user and a coordinates correction position S thereof, the steelframe T that becomes a reference for correction is identified.

FIG. 2(b) is a view illustrating a steel frame T2 that is the nearestfrom a user U when the user U wearing an image display device 100, whichrepresents MR goggles, moves and coordinates correction positions S21 toS24. In this embodiment, the image display device 100 can perceive aposition of the user U with respect to the steel frame T, that is, arelative positional relation thereof by using a known self-positionestimation technology.

In accordance with the user U seeing the steel frame T2, the imagedisplay device 100 determines one of the coordinates correctionpositions S21 to S24 of the steel frame T2 that is the nearest from theuser U as a correction reference. As illustrated in FIG. 3(a), the imagedisplay device 100 selects a coordinates correction position S at whicha reverse direction vector of a visual line direction of the user U(here, corresponding to an imaging direction of a camera in the imagedisplay device) and a normal line vector of a surface of the steel frameT are within a predetermined angle.

In addition, an actual coordinates correction position that is definedon the face of the steel frame T is calculated on the basis of adistance to the steel frame T measured using a depth sensor (FIG. 3(b)).The depth sensor may not always be able to measure from in front of thesteel frame T. Thus, the coordinates correction position is corrected toa predetermined position (for example, a center part) on the surface ofthe steel frame T.

Then, a process of shifting a correction correspondence part Sh2 of theimage data G is performed on the basis of the coordinates correctionposition (FIG. 3(c)). Image data G2 is associated with correctioncorrespondence part information associated with each coordinatescorrection position S (a position S22 in FIG. 3 ) and relativepositional information in the image data G. In other words, each face ofeach piece of image data G is associated with a coordinates correctionposition set in each face of each steel frame T and a reference positionthat is used for position alignment. By aligning the position of thiscorrection correspondence part Sh2 with the coordinates correctionposition S12, the image data G and the steel frame T can be aligned inposition.

By performing such a process, a user U can visually recognize a state inwhich the steel frame T is covered with the image data G of thedecorative material while moving. In FIGS. 2 and 3 , although thecross-section of the steel frame T is configured in a quadrangle shape,the configuration is not limited thereto. The cross-section may beconfigured in the shape of a circle or another polygon, and thecoordinates correction position of the steel frame T and the correctioncorrespondence part of the image data G of the decorative material maybe associated with each other.

Next, a functional configuration of the image display device 100according to this embodiment will be described. FIG. 4 is a blockdiagram illustrating the functional configuration of the image displaydevice 100. As illustrated in the drawing, the image display device 100is configured to include a camera 101, a correction processing unit 102(a correction unit), a storage unit 103, a display unit 104, a depthsensor 105 (a sensor unit), a computation processing unit 106, and aself-position/posture estimating unit 107. This image display device 100is a device that is so-called MR goggles and is a device of ahead-mounted display type. By wearing this image display device 100 onthe head, a user can see a video captured by the camera 101 through thedisplay unit 104. In a head-mount display of a video see-through type,by wearing this image display device 100 on the head, a user can see avideo captured by the camera 101 through the display unit 104. Thisembodiment is not limited to these forms and can be similarly realizedusing a device of an optical see through type (for example, steel framesare seen through glass).

The camera 101 is a part that images a reality space seen by a user.

The correction processing unit 102 is a part that composes image datathat is a virtual space member with a video captured by the camera 101in a composite reality space. Here, the correction processing unit 102performs a process for composing image data G representing decorativematerials with steel frames T and a correction process at the time of apositional deviation.

The storage unit 103 is a part that stores design data of structuressuch as steel frames T and image data G representing decorativematerials for the structures. The design data is informationrepresenting a size of each steel frame T, a gap between steel frames,and the like. In addition, the image data G has information representinga correspondence relation with a steel frame T (particularly, acoordinates correction position and the like) in addition to the imageof decorative materials.

The display unit 104 is a part that displays a video captured by thecamera 101. In a case in which the image data G is composed with thesteel frames T by the correction processing unit 102, the display unit104 displays the composed video.

The depth sensor 105 is a sensor that measures a distance between a userU and a steel frame T.

The computation processing unit 106 is a part that calculates an initialposition of a relative positional relation of the image display device100 with respect to the structure (the steel frame T) and a positionaldeviation between the steel frame T and the image data G that isgenerated when the user moves on the basis of the design data stored inthe storage unit 103.

The self-position/posture estimating unit 107 is a part that estimates aposition in a reality space that is captured and displayed by the camera101 and a change in a posture of the image display device 100 on thebasis of a video captured by the camera 101. By perceiving a directionin which the steel frame T is visually recognized and a relativedistance thereof from a captured video on the basis of the initialposition calculated by the computation processing unit 106, an image ofan initial marker at that time, and the steel frame T that is beingcurrently captured, the self-position/posture estimating unit 107 canestimate a current positional relation (a current position) relative tothe steel frame T. This self-position/posture estimating unit 107 can berealized using a known technology, for example, simultaneouslocalization and mapping (SLAM). The self-position/posture estimationfunction is not limited thereto, and self-position estimation may beperformed using a GPS or any other sensor.

Operations of the image display device 100 configured in this way willbe described. FIG. 5 is a flowchart illustrating the operations.

The computation processing unit 106 calculates a coordinates correctionposition S in each steel frame T on the basis of design data stored inthe storage unit 103 and stores the calculated coordinates correctionposition in the storage unit 103 (S101).

Next, the computation processing unit 106 calculates an initial positionof the self-position/posture estimating unit 107 and perceives apositional relation of the steel frame T (S102).

When a user moves (S103), an error occurs in the self-position/positionestimation (S104).

Then, the correction processing unit 102 selects a coordinatescorrection position that becomes a reference from coordinates correctionposition candidates on the basis of the self-position estimated by theself-position/posture estimating unit 107. Then, the correctionprocessing unit 102 corrects the position of the image data G of thedecorative material on the basis of a result of the self-positionestimation (the self-position) and the selected coordinates correctionposition (S105). These S103 to S106 are repeated until the program ends(S106).

FIG. 6 is a flowchart illustrating detailed operations of a processS101. The computation processing unit 106 reads design data of abuilding (the steel frame T) by referring to the storage unit 103(S201). The computation processing unit 106 calculates a coordinatescorrection position from floor height information and base lineinformation (S202). Here, the floor height information represents thenumber of stories and a height of a building. The base line informationis information that indicates a center line of the steel frame andrepresents coordinates inside the building. The coordinates correctionposition is a position the becomes a reference for composing the imagedata G of a decorative material and, here, represents three-dimensionalcoordinates obtained from coordinates of a coordinates correctionposition defined at an intersection C of the base line and a heightthereof.

The computation processing unit 106 sets a position acquired by shiftingthe coordinates correction position by Y meters in a height direction asa coordinates correction position (S203). This process is for thepurpose of alignment with a position aligned with the eye line of aperson.

The computation processing unit 106 determines whether or not the finalcoordinates correction position is present in a steel frame such as awall or a column (S204). For example, it is determined that coordinatesof an intersection of a base line (coordinates in a horizontaldirection) are present inside the steel frame T. This similarly appliesalso to a case in which the steel frame T is buried in a wall. Suchinformation is determined on the basis of sizes of steel frames T and anarrangement thereof included in the design data.

In a case in which it is determined that the coordinates correctionposition is present in the steel frame T, the computation processingunit 106 splits the coordinates correction position in four directionsusing the intersection of the base line set as a center, and calculatesand sets intersections intersecting with the surface of the steel frameT in the design data as coordinates correction positions (S205).

The computation processing unit 106 stores the coordinates correctionpositions calculated on the basis of these processes in the storage unit103 (S206). The storage unit 103 stores, for each of the steel frames T(the steel frames T1 to Tn), identification information of a decorativematerial associated therewith, an arrangement position (relativeposition information) of the decorative material, a correctioncorrespondence part in image data G of the decorative materialcorresponding to the coordinates correction position set for each steelframe T, and image data G (image data G1 to Gn) of the decorativematerial in association with each other. In other words, the storageunit 103 stores various kinds of information such that image data G (G1to Gn) to be composed is associated with each coordinates correctionposition of the steel frame T.

Next, detailed processes of the process S105 will be described. FIG. 7is a flowchart illustrating detailed processes of a process of selectinga self-position/posture estimation result and a correction processperformed by the correction processing unit 102.

The correction processing unit 102 selects a coordinates correctionposition that becomes a reference from among coordinates correctionposition candidates stored in the storage unit 103 (S301). In a case inwhich a candidate is present, the correction processing unit 102superimposes message information indicating a coordinates correctionposition on the basis of the selected coordinates correction position,and the display unit 104 displays an indication thereof (S303).

The correction processing unit 102 calculates a deviation of thecoordinates correction position with coordinates in an actual space(S304). In other words, the correction processing unit 102 calculates apositional deviation between the coordinates correction position in thedesignated steel frame T and the image data G of a decorative materialthat is a composition target. For example, in FIG. 3(b), the depthsensor 105 measures a distance from a position obtained by shifting theheight of the intersection C2 of the base line by Y to the surface (thecoordinates correction position) of the steel frame T. The correctionprocessing unit 102 calculates an actual position of the coordinatescorrection position on the basis of the distance. On the other hand, thecorrection processing unit 102 acquires a position (a position to bealigned with the coordinates correction position; here, the correctioncorrespondence part) of the image data G2 of the decorative material andcalculates a difference thereof as a positional deviation (error). Thecoordinates correction position is located at the position acquiredthrough shift by Y in the height direction and at the center position ofthe thickness of the steel frame in the horizontal direction, and thusthe position may be acquired through calculation or may be identified inaccordance with a user's instruction using a user input functionincluded in this device.

In a case in which a positional deviation (the deviation is equal to orlarger than a predetermined value) is calculated (S305: Yes), thecorrection processing unit 102 corrects a superimposition position ofthe image data G in accordance with the deviation amount (S306).

Next, detailed processes of the process S301 represented in FIG. 7 willbe described. FIG. 8 is a flowchart illustrating a process of selectinga coordinates correction position to be used from coordinates correctionposition candidates.

The correction processing unit 102 reads a coordinates correctionposition in the steel frame T from the storage unit 103 (S401) andacquires a result of self-position/posture estimation (relativeself-position information to the steel frame T) performed by theself-position/posture estimating unit 107 (S402).

The correction processing unit 102 extracts a plurality of coordinatescorrection positions S present near (in a predetermined range) of theself-position on the basis of the result of the self-position/postureestimation (S403). In addition, the correction processing unit 102calculates a direction N1 in which a user faces the steel frame T on thebasis of the result of the self-position/posture estimation (S404). Inother words, the self-position/posture estimating unit 107 calculates anangle formed between a direction vector of a normal line N2 of thesurface of each steel frame T and a reverse direction vector of thedirection N1.

The correction processing unit 102 excludes faces each having acoordinates correction position at which this angle is equal to orlarger than y degrees. In other words, in a case in which the surfacesof steel frames T are seen in an inclining direction by a user,coordinates correction positions on the surfaces of the steel frames Tare excluded (S405).

The correction processing unit 102 determines whether or not a distancebetween a coordinates correction position S that is the nearest and acoordinates correction position S that is the second nearest is shorterthan a predetermined distance (S406). The correction processing unit 102determines that there is no candidate in a case in which the distance isshorter than the predetermined distance (S408) and selects the nearestcoordinates correction position S in a case in which the distance isequal to or longer than the predetermined distance (S407).

In accordance with this process, the correction processing unit 102selects one coordinates correction position S.

Next, detailed processes of the process S304 represented in FIG. 7 willbe described. FIG. 9 is a flowchart illustrating a process ofcalculating a deviation of a coordinates correction position that isperformed by the correction processing unit 102. FIG. 9 illustrates aprocess using a reference marker for each part of a steel frame T.

In a case in which a reference marker is shown in an image captured bythe camera 101, the correction processing unit 102 calculates a positionP1 of the reference marker on the basis of the self-position estimatedby the self-position/posture estimating unit 107 (S501 and S502). Thecorrection processing unit 102 has information such as shapes, sizes,and the like of reference markers in advance and can determine adirection and a degree of distance from which capturing has beenperformed on the basis of a video of the reference markers captured bythe camera 101.

In addition, the correction processing unit 102 acquires a position P2that is a correction correspondence part corresponding to a coordinatescorrection position of the steel frame T in image data Gn of adecorative material that is being currently displayed (S503). Thecorrection processing unit 102 perceives a correction correspondencepart corresponding to a coordinates correction position of a steel frameT and a position thereof in each piece of image data Gn and acquiresposition information of the correction correspondence part correspondingto the coordinates correction position in the image data Gn to compose aplace near the position P1.

The correction processing unit 102 calculates a difference between theposition P1 and the position P2 as an estimation error (S504).

In accordance with movement of a user, a deviation between the imagedata G and the steel frames T occurs. The reason for this is that,although the image data G and the steel frames T, first, are alignedwith each other in position using the initial marker, it is difficult tocompletely align the positions into details, and a position/postureestimation error of SLAM is accumulated in accordance with the movement.Thus, there are cases in which the deviation becomes large in accordancewith the movement of a user. The process described above is a processperformed when the deviation is calculated.

Although FIG. 9 described above illustrates the process using referencemarkers, the reference markers may not be used. FIG. 10 is a flowchartillustrating another process performed when a positional deviationbetween image data G and steel frames T is calculated.

When an input of coordinates of a coordinates correction position isaccepted (S601: Yes), the depth sensor 105 calculates a distance to thecoordinates correction position on a steel frame T that is present in acapturing direction of the camera 101 (a sight line direction of a user)and calculates a position P1 (S602). In addition, the correctionprocessing unit 102 acquires a position P2 that is a correctioncorrespondence part corresponding to a coordinates correction positionof the steel frame T in the image data G that is being currentlydisplayed (S603). The correction processing unit 102 calculates adifference between the position P1 and the position P2 as an estimationerror (S604).

In this way, without using reference markers, the position P1 can becalculated, and a positional deviation on the basis of the position P1can be calculated.

Next, operations and effects according to one aspect of this embodimentwill be described. The image display device 100 according to one aspectof this embodiment is a device that performs imaging using the camera101 and composes predetermined image data G (including image data G1 toGn) with a steel frame T that is a composition target object havingsteel frames T1 to Tn that are a plurality of partial members present inthe field of view of a user and displays a result of the composition.This image display device 100 includes the storage unit 103 that storesdesign data of a steel frame T that is a composition target object, thedisplay unit 104 that generates a composed image acquired by composingimage data G representing a predetermined decorative material on thebasis of a reference position (a reference marker or a coordinatescorrection position) in one part of the steel frame T (for example, asteel frame T1 or the like) and displays the composed image, and thecorrection processing unit 102 that corrects a deviation when thedeviation occurs between the image data G and the steel frame T in thefield of view of a user U in accordance with movement of the user.

This correction processing unit 102 acquires a coordinates correctionposition S that becomes a reference position used for a correctionprocess from the steel frame T1 and the like that are partial membersusing design data stored by the storage unit 103 and a self-position ofthe user U with respect to the steel frame T.

The correction processing unit 102 performs position alignment of imagedata G on the basis of the one coordinates correction position S,thereby performing correction.

In accordance with this process, a steel frame Tn can be identifiedwithout attaching markers corresponding to parts (steel frames T1 to Tn)of the steel frame T and can compose image data Gn representing adecorative material corresponding to the steel frame Tn.

In this image display device 100, when a coordinates correction positionthat becomes a reference position used for a correction process isacquired, the correction processing unit 102 acquires a plurality ofcoordinates correction positions and selects one coordinates correctionposition satisfying a predetermined condition (for example, beingnearest) for the self-position of the user U from among the plurality ofcoordinates correction positions.

In this way, a more appropriate coordinates correction position can beselected.

In addition, in this embodiment, the predetermined image data Grepresents a decorative material (a coating member) configure to coverthe steel frame T that is a composition target object.

In accordance with this, position alignment between the steel frame Tand the image data G can be performed, and a user can visually recognizea building including the decorative material.

In this embodiment, the steel frame T is composed of a plurality offaces. For example, the cross-section thereof has a tetrahedron. Thecoordinates correction positions are defined on the plurality of facesthereof. When one coordinates correction position S is selected, thecorrection processing unit 102 selects a coordinates correction positiondefined on a face having a predetermined angle with respect to adirection for each surface of the steel frame T from the self-positionof the user U among surfaces of the plurality of steel frames T.

In this way, a more appropriate coordinates correction position can beselected, and thus composition of image data can be accuratelyperformed.

In this image display device 100, the correction processing unit 102selects a coordinates correction position that is the nearest from theuser U as one coordinates correction position. In this way, anappropriate coordinates correction position can be selected.

In addition, this image display device 100 further includes the depthsensor 105 that measures a distance to the steel frame T. The correctionprocessing unit 102 calculates an actual coordinates correction positionusing the distance measured by the depth sensor 105. Then, thecorrection processing unit 102 calculates a deviation between the actualcoordinates correction position and a position corresponding to thecoordinates correction position S in the image data G composed beforecorrection and performs correction on the basis of the deviation.

In addition, the image display device 100 further includes a recognitionunit (not illustrated) that recognizes a marker attached to acomposition target object. The correction processing unit 102 calculatesan actual coordinates correction position on the basis of the positionof the recognized marker. Then, the correction processing unit 102calculates a deviation between the actual coordinates correctionposition and a position corresponding to the coordinates correctionposition in the image data G composed before correction and performscorrection on the basis of this deviation.

Correction can be performed on the basis of these deviations.

The block diagram used for the description of the above embodimentsshows blocks of functions. Those functional blocks (component parts) areimplemented by any combination of at least one of hardware and software.Further, a means of implementing each functional block is notparticularly limited. Specifically, each functional block may beimplemented by one physically or logically combined device or may beimplemented by two or more physically or logically separated devicesthat are directly or indirectly connected (e.g., by using wired orwireless connection etc.). The functional blocks may be implemented bycombining software with the above-described one device or theabove-described plurality of devices.

The functions include determining, deciding, judging, calculating,computing, processing, deriving, investigating, lookingup/searching/inquiring, ascertaining, receiving, transmitting,outputting, accessing, resolving, selecting, choosing, establishing,comparing, assuming, expecting, considering, broadcasting, notifying,communicating, forwarding, configuring, reconfiguring,allocating/mapping, assigning and the like, though not limited thereto.For example, the functional block (component part) that implements thefunction of transmitting is referred to as a transmitting unit or atransmitter. In any case, a means of implementation is not particularlylimited as described above.

For example, the image display device 100 according to one embodiment ofthe present disclosure may function as a computer that performsprocessing of an image process method in an image process according tothe present disclosure. FIG. 10 is a view showing an example of thehardware configuration of the image display device 100 according to oneembodiment of the present disclosure. The image display device 100described above may be physically configured as a computer device thatincludes a processor 1001, a memory 1002, a storage 1003, acommunication device 1004, an input device 1005, an output device 1006,a bus 1007 and the like.

In the following description, the term “device” may be replaced with acircuit, a device, a unit, or the like. The hardware configuration ofthe image display device 100 may be configured to include one or aplurality of the devices shown in the drawings or may be configuredwithout including some of those devices.

The functions of the image display device 100 may be implemented byloading predetermined software (programs) on hardware such as theprocessor 1001 and the memory 1002, so that the processor 1001 performscomputations to control communications by the communication device 1004and control at least one of reading and writing of data in the memory1002 and the storage 1003.

The processor 1001 may, for example, operate an operating system tocontrol the entire computer. The processor 1001 may be configured toinclude a CPU (Central Processing Unit) including an interface with aperipheral device, a control device, an arithmetic device, a registerand the like. For example, the correction processing unit 101, thecomputation processing unit 106, elf-position/posture estimating unit107, and the like described above may be implemented by the processor1001.

Further, the processor 1001 loads a program (program code), a softwaremodule and data from at least one of the storage 1003 and thecommunication device 1004 into the memory 1002 and performs variousprocessing according to them. As the program, a program that causes acomputer to execute at least some of the operations described in theabove embodiments is used. For example, the correction processing unit102 in the image display device 100 and the like may be implemented by acontrol program that is stored in the memory 1002 and operates on theprocessor 1001, and the other functional blocks may be implemented inthe same way. Although the above-described processing is executed by oneprocessor 1001 in the above description, the processing may be executedsimultaneously or sequentially by two or more processors 1001. Theprocessor 1001 may be implemented in one or more chips. Note that theprogram may be transmitted from a network through a telecommunicationsline.

The memory 1002 is a computer-readable recording medium, and it may becomposed of at least one of ROM (Read Only Memory), EPROM (ErasableProgrammable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM(Random Access Memory) and the like, for example. The memory 1002 may bealso called a register, a cache, a main memory (main storage device) orthe like. The memory 1002 can store a program (program code), a softwaremodule and the like that can be executed for implementing an imageprocessing method according to one embodiment of the present disclosure.

The storage 1003 is a computer-readable recording medium, and it may becomposed of at least one of an optical disk such as a CD-ROM (CompactDisk ROM), a hard disk drive, a flexible disk, a magneto-optical disk(e.g., a compact disk, a digital versatile disk, and a Blu-ray(registered trademark) disk), a smart card, a flash memory (e.g., acard, a stick, and a key drive), a floppy (registered trademark) disk, amagnetic strip and the like, for example. The storage 1003 may be calledan auxiliary storage device. The above-described storage medium may be adatabase, a server, or another appropriate medium including the memory1002 and/or the storage 1003, for example.

The communication device 1004 is hardware (a transmitting and receivingdevice) for performing communication between computers via at least oneof a wired network and a wireless network, and it may also be referredto as a network device, a network controller, a network card, acommunication module, or the like. The communication device 1004 mayinclude a high-frequency switch, a duplexer, a filter, a frequencysynthesizer or the like in order to implement at least one of FDD(Frequency Division Duplex) and TDD (Time Division Duplex), for example.

The input device 1005 is an input device (e.g., a keyboard, a mouse, amicrophone, a switch, a button, a sensor, etc.) that receives an inputfrom the outside. The output device 1006 is an output device (e.g., adisplay, a speaker, an LED lamp, etc.) that makes output to the outside.Note that the input device 1005 and the output device 1006 may beintegrated (e.g., a touch panel).

In addition, the devices such as the processor 1001 and the memory 1002are connected by the bus 1007 for communicating information. The bus1007 may be a single bus or may be composed of different buses betweendifferent devices.

Further, the image display device 100 may include hardware such as amicroprocessor, a DSP (Digital Signal Processor), an ASIC (ApplicationSpecific Integrated Circuit), a PLD (Programmable Logic Device), and anFPGA (Field Programmable Gate Array), and some or all of the functionalblocks may be implemented by the above-described hardware components.For example, the processor 1001 may be implemented with at least one ofthese hardware components.

The procedure, the sequence, the flowchart and the like in each of theaspects/embodiments described in the present disclosure may be in adifferent order unless inconsistency arises. For example, for the methoddescribed in the present disclosure, elements of various steps aredescribed in an exemplified order, and it is not limited to the specificorder described above.

Input/output information or the like may be stored in a specificlocation (e.g., memory) or managed in a management table. Further,input/output information or the like can be overwritten or updated, oradditional data can be written. Output information or the like may bedeleted. Input information or the like may be transmitted to anotherdevice.

The determination may be made by a value represented by one bit (0 or1), by a truth-value (Boolean: true or false), or by numericalcomparison (e.g., comparison with a specified value).

Each of the aspects/embodiments described in the present disclosure maybe used alone, may be used in combination, or may be used by beingswitched according to the execution. Further, a notification ofspecified information (e.g., a notification of “being X”) is not limitedto be made explicitly, and it may be made implicitly (e.g., anotification of the specified information is not made).

Although the present disclosure is described in detail above, it isapparent to those skilled in the art that the present disclosure is notrestricted to the embodiments described in this disclosure. The presentdisclosure can be implemented as a modified and changed form withoutdeviating from the spirit and scope of the present disclosure defined bythe appended claims. Accordingly, the description of the presentdisclosure is given merely by way of illustration and does not have anyrestrictive meaning to the present disclosure.

Software may be called any of software, firmware, middleware, microcode,hardware description language or another name, and it should beinterpreted widely so as to mean an instruction, an instruction set, acode, a code segment, a program code, a program, a sub-program, asoftware module, an application, a software application, a softwarepackage, a routine, a sub-routine, an object, an executable file, athread of execution, a procedure, a function and the like.

Further, software, instructions and the like may be transmitted andreceived via a transmission medium. For example, when software istransmitted from a website, a server or another remote source using atleast one of wired technology (a coaxial cable, an optical fiber cable,a twisted pair and a digital subscriber line (DSL) etc.) and wirelesstechnology (infrared rays, microwave etc.), at least one of those wiredtechnology and wireless technology are included in the definition of thetransmission medium.

The information, signals and the like described in the presentdisclosure may be represented by any of various different technologies.For example, data, an instruction, a command, information, a signal, abit, a symbol, a chip and the like that can be referred to in the abovedescription may be represented by a voltage, a current, anelectromagnetic wave, a magnetic field or a magnetic particle, anoptical field or a photon, or an arbitrary combination of them.

Note that the term described in the present disclosure and the termneeded to understand the present disclosure may be replaced by a termhaving the same or similar meaning. For example, at least one of achannel and a symbol may be a signal (signaling). Further, a signal maybe a message. Furthermore, a component carrier (CC) may be called acell, a frequency carrier, or the like.

The terms “system” and “network” used in the present disclosure are usedto be compatible with each other.

Information, parameters, and the like described in the presentdisclosure may be represented using absolute values, may be representedusing relative values from predetermined values, or may be representedusing other information corresponding thereto. For example, a wirelessresource may be indicated using an index.

The names used for the above-described parameters are not definitive inany way.

Note that the term “determining” and “determining” used in the presentdisclosure includes a variety of operations. For example, “determining”and “determining” can include regarding the act of judging, calculating,computing, processing, deriving, investigating, lookingup/searching/inquiring (e.g., looking up in a table, a database oranother data structure), ascertaining or the like as being “determined”and “determined”. Further, “determining” and “determining” can includeregarding the act of receiving (e.g., receiving information),transmitting (e.g., transmitting information), inputting, outputting,accessing (e.g., accessing data in a memory) or the like as being“determined” and “determined”. Further, “determining” and “determining”can include regarding the act of resolving, selecting, choosing,establishing, comparing or the like as being “determined” and“determined”. In other words, “determining” and “determining” caninclude regarding a certain operation as being “determined” and“determined”. Further, “determining (determining)” may be replaced with“assuming”, “expecting”, “considering” and the like.

The term “connected”, “coupled” or every transformation of this termmeans every direct or indirect connection or coupling between two ormore elements, and it includes the case where there are one or moreintermediate elements between two elements that are “connected” or“coupled” to each other. The coupling or connection between elements maybe physical, logical, or a combination of them. For example, “connect”may be replaced with “access”. When used in the present disclosure, itis considered that two elements are “connected” or “coupled” to eachother by using at least one of one or more electric wires, cables, andprinted electric connections and, as several non-definitive andnon-comprehensive examples, by using electromagnetic energy such aselectromagnetic energy having a wavelength of a radio frequency region,a microwave region and an optical (both visible and invisible) region.

The description “on the basis of” used in the present disclosure doesnot mean “only on the basis of” unless otherwise noted. In other words,the description “on the basis of” means both of “only on the basis of”and “at least on the basis of”.

Furthermore, “means” in the configuration of each device described abovemay be replaced by “unit”, “circuit”, “device” or the like.

As long as “include”, “including” and transformation of them are used inthe present disclosure, those terms are intended to be comprehensivelike the term “comprising”. Further, the term “or” used in the presentdisclosure is intended not to be exclusive OR.

In the present disclosure, when articles, such as “a”, “an”, and “the”in English, for example, are added by translation, the presentdisclosure may include that nouns following such articles are plural.

In the present disclosure, the term “A and B are different” may meanthat “A and B are different from each other”. Note that this term maymean that “A and B are different from C”. The terms such as “separated”and “coupled” may be also interpreted in the same manner.

REFERENCE SIGNS LIST

-   -   100 Image display device    -   101 Camera    -   102 correction processing unit    -   103 storage unit    -   104 display unit    -   105 depth sensor    -   106 computation processing unit    -   107 self-position/posture estimating unit

1. An image display device that composes and displays a predeterminedimage for a composition target object present in a reality space, theimage display device comprising: a storage unit configured to storedesign data representing a positional relation between parts of thecomposition target object; a display unit configured to generate anddisplays a composed image acquired by composing a predetermined imageusing one part of the composition target object as a reference position;and a correction unit configured to correct a deviation between thepredetermined image and the composition target object that occurs inaccordance with movement of a user, wherein the correction unit acquiresa coordinates correction position that becomes a reference position usedfor a correction process in the one part of the composition targetobject using the design data and a self-position of the user withrespect to the composition target object and corrects a compositionposition of the predetermined image on the basis of the coordinatescorrection position.
 2. The image display device according to claim 1,wherein, when the coordinates correction position that becomes thereference position used for the correction process is acquired, thecorrection unit acquires a plurality of coordinates correction positionsand selects one coordinates correction position satisfying apredetermined condition for the self-position of the user from theplurality of coordinates correction positions.
 3. The image displaydevice according to claim 1, wherein the predetermined image representsa coating member configured to cover the composition target object. 4.The image display device according to claim 2, wherein, when the onecoordinates correction position is selected, the correction unit selectsa coordinates correction position defined in a composition target objecthaving a predetermined angle with respect to a direction for eachcomposition target object from the self-position of the user among aplurality of composition target objects.
 5. The image display deviceaccording to claim 4, wherein the correction unit selects a coordinatescorrection position that is the nearest from the user as the onecoordinates correction position.
 6. The image display device accordingto claim 1, further comprising a sensor unit configured to measure adistance to the composition target object, wherein the correction unitcalculates an actual coordinates correction position using the distancemeasured by the sensor unit, calculates a deviation between the actualcoordinates correction position and a part corresponding to thecoordinates correction position in an image composed before correction,and performs correction on the basis of the deviation.
 7. The imagedisplay device according to claim 1, further comprising a recognitionunit configured to recognize a marker attached to a composition targetobject, wherein the correction unit calculates an actual coordinatescorrection position on the basis of a position of the recognized marker,calculates a deviation between the actual coordinates correctionposition and a part corresponding to the coordinates correction positionin an image composed before correction, and performs correction on thebasis of the deviation.
 8. The image display device according to claim2, wherein the predetermined image represents a coating memberconfigured to cover the composition target object.
 9. The image displaydevice according to claim 2, further comprising a sensor unit configuredto measure a distance to the composition target object, wherein thecorrection unit calculates an actual coordinates correction positionusing the distance measured by the sensor unit, calculates a deviationbetween the actual coordinates correction position and a partcorresponding to the coordinates correction position in an imagecomposed before correction, and performs correction on the basis of thedeviation.
 10. The image display device according to claim 3, furthercomprising a sensor unit configured to measure a distance to thecomposition target object, wherein the correction unit calculates anactual coordinates correction position using the distance measured bythe sensor unit, calculates a deviation between the actual coordinatescorrection position and a part corresponding to the coordinatescorrection position in an image composed before correction, and performscorrection on the basis of the deviation.
 11. The image display deviceaccording to claim 4, further comprising a sensor unit configured tomeasure a distance to the composition target object, wherein thecorrection unit calculates an actual coordinates correction positionusing the distance measured by the sensor unit, calculates a deviationbetween the actual coordinates correction position and a partcorresponding to the coordinates correction position in an imagecomposed before correction, and performs correction on the basis of thedeviation.
 12. The image display device according to claim 5, furthercomprising a sensor unit configured to measure a distance to thecomposition target object, wherein the correction unit calculates anactual coordinates correction position using the distance measured bythe sensor unit, calculates a deviation between the actual coordinatescorrection position and a part corresponding to the coordinatescorrection position in an image composed before correction, and performscorrection on the basis of the deviation.
 13. The image display deviceaccording to claim 2, further comprising a recognition unit configuredto recognize a marker attached to a composition target object, whereinthe correction unit calculates an actual coordinates correction positionon the basis of a position of the recognized marker, calculates adeviation between the actual coordinates correction position and a partcorresponding to the coordinates correction position in an imagecomposed before correction, and performs correction on the basis of thedeviation.
 14. The image display device according to claim 3, furthercomprising a recognition unit configured to recognize a marker attachedto a composition target object, wherein the correction unit calculatesan actual coordinates correction position on the basis of a position ofthe recognized marker, calculates a deviation between the actualcoordinates correction position and a part corresponding to thecoordinates correction position in an image composed before correction,and performs correction on the basis of the deviation.
 15. The imagedisplay device according to claim 4, further comprising a recognitionunit configured to recognize a marker attached to a composition targetobject, wherein the correction unit calculates an actual coordinatescorrection position on the basis of a position of the recognized marker,calculates a deviation between the actual coordinates correctionposition and a part corresponding to the coordinates correction positionin an image composed before correction, and performs correction on thebasis of the deviation.
 16. The image display device according to claim5, further comprising a recognition unit configured to recognize amarker attached to a composition target object, wherein the correctionunit calculates an actual coordinates correction position on the basisof a position of the recognized marker, calculates a deviation betweenthe actual coordinates correction position and a part corresponding tothe coordinates correction position in an image composed beforecorrection, and performs correction on the basis of the deviation.